Improve generated adversarial imitation learning with reward variance regularization

Imitation learning aims at recovering expert policies from limited demonstration data. Generative Adversarial Learning (GAIL) employs the generative adversarial framework for imitation and has shown great potentials. GAIL its variants, however, are found highly sensitive to hyperparameters hard converge well in practice. One key issue is that supervised discriminator a much faster speed than reinforcement generator, making generator gradient vanishing. Although formulated as zero-sum game, ultimate goal of learn thus should play role more like teacher rather real opponent. Therefore, consider how could learn. In this paper, we disclose enhancing training equivalent increase variance fake reward provided by output. We propose an improved version GAIL, GAIL-VR, which also learns avoid vanishing through regularization rewards variance. Experiments various tasks, including locomotion tasks Atari games, indicate GAIL-VR can improve stability scores.